Automated configuration of behavior of a telepresence system based on spatial detection of telepresence components

ABSTRACT

A system that automatically configures the behavior of the display devices of a video conference endpoint. The controller may detect, at a microphone array having a predetermined physical relationship with respect to a camera, audio emitted from one or more loudspeakers, each loudspeaker having a predetermined physical relationship with respect to at least one of one or more display devices in a conference room. The controller may then generate data representing a spatial relationship between the one or more display devices and the camera based on the detected audio. Finally, the controller may assign video sources received by the endpoint to each of the one or more display devices based on the data representing the spatial relationship and the content of each received video source, and may also assign outputs from multiple video cameras to an outgoing video stream based on the on the data representing the spatial relationship.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.15/338,676, filed on Oct. 31, 2016, and entitled “AUTOMATEDCONFIGURATION OF BEHAVIOR OF A TELEPRESENCE SYSTEM BASED ON SPATIALDETECTION OF TELEPRESENCE COMPONENTS,” the entirety of which isincorporated herein by reference.

TECHNICAL FIELD

The present disclosure relates to configuring components of a videoconference endpoint in a conference room based on spatial detection ofthe components.

BACKGROUND

Video conference endpoints are deployed in conference rooms. Theconference rooms can differ in size and configuration which affects thelayout/placement of the video conference endpoint components in theconference room, and use of the conference room. The placement of thecomponents within the conference room, such as the relationship andplacement of the display screens with respect to the camera(s), affectsthe experience of participants of a conference session within theconference room. Because no two conference rooms are the same size andshape, a standard layout for a conference room is not possible. Thisresults in different placements of the camera(s) with respect to thedisplay screens of a conference room. Typically, an operator has tomanually select which display screen is to receive specific videosources, including which display screen, or portion of a display screen,is to display the live video stream of the participants of theconference session that are present at another video conferenceendpoint. Such manual selection is cumbersome and inconvenient, andoften does not place the live video stream of participants of theconference session from another video conference endpoint at a positionthat maximizes eye contact between participants at separate videoconference endpoints.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of an example video conference (e.g.,teleconference) system in which techniques to automatically configurethe behavior of various components within the environment based onspatial detection may be implemented, according to an exampleembodiment.

FIG. 2A is an illustration of an example video conference endpointdeployed in a conference room and configured to perform techniquespresented herein, according to an example embodiment.

FIG. 2B is an illustration of example video conference endpointsdeployed in respective conference rooms and configured to performtechniques presented herein, according to an example embodiment.

FIG. 3A is a front view of one of the display devices of a videoconference endpoint where the integrated camera and microphone array aredisposed above the display device, according to an example embodiment.

FIG. 3B is a front view of one of the display devices of a videoconference endpoint where the integrated camera and microphone array aredisposed below the display device, according to an example embodiment.

FIG. 3C is a front view of one of the display devices of a videoconference endpoint where the integrated camera and microphone array aredisposed to the right of the display device, according to an exampleembodiment.

FIG. 3D is a front view of one of the display devices of a videoconference endpoint where the integrated camera and microphone array aredisposed to the left of the display device, according to an exampleembodiment.

FIG. 4A is a front view of a plurality of display devices of a videoconference endpoint where the integrated camera and microphone array aredisposed above one of the display devices, according to an exampleembodiment.

FIG. 4B is a front view of a plurality of display devices of a videoconference endpoint where the integrated camera and microphone array aredisposed between the display devices, according to an exampleembodiment.

FIG. 5 is a block diagram of an example controller of a video conferenceendpoint configured to perform techniques described herein, according toan embodiment.

FIG. 6 is an illustration of an example user control device associatedwith a video conference endpoint, where the user control device displaysa rendering of the components of the video conference endpoint.

FIG. 7A is a front view of a camera integrated with a microphone arrayof a video conference endpoint, where the microphone array is detectingaudio outputs originating from either the left side or the right side ofthe microphone array, according to an example embodiment.

FIG. 7B is a front view of a camera integrated with a microphone arrayof the video conference endpoint where the microphone array is detectingaudio outputs originating from either above or below the microphonearray, according to an example embodiment.

FIG. 8 is a flowchart of a method of updating the display device thatreceives a live video feed based on the determined probability an audiooutput originates from above, below, to the right of, and to the left ofthe microphone array, according to an example embodiment.

FIG. 9 is a flowchart of a method for configuring the roles of aplurality of display devices of a conference room based on detectedaudio outputs of the plurality of display devices, according to anexample embodiment.

FIG. 10 is a flowchart of a method for configuring the roles of aplurality of display devices of a conference room based on detectedaudio outputs of the plurality display devices, the detected shape ofthe table of the conference endpoint, and/or the orientation of thedetected faces within the conference room, according to an exampleembodiment.

FIG. 11A is an illustration of a table and a plurality of detected faceswithin a conference room from the viewpoint of a camera integrated witha video conference endpoint where the camera is mounted below a displaydevice, according to an example embodiment.

FIG. 11B is another illustration of a table and a plurality of detectedfaces within a conference room from the viewpoint of a camera integratedwith a video conference endpoint of where the camera is mounted at thesame height as a display device, according to an example embodiment.

FIG. 11C is an illustration of a table and a plurality of detected faceswithin a conference room from the viewpoint of a camera integrated witha video conference endpoint of where the camera is mounted above adisplay device, according to an example embodiment.

FIG. 12 is a flowchart of a method of generating data representing thespatial relationship of the components of the video conference endpoint,according to an example embodiment.

DESCRIPTION OF EXAMPLE EMBODIMENTS Overview

Techniques presented herein relate to automatically configuring the oneor more display devices of a video conference endpoint based on spatialdetection of the components of the video conference endpoint and thecontent of the video sources received by the video conference endpoint.The video conference endpoint may include one or more display devices,one or more loudspeakers having a predetermined physical relationshipwith respect to at least one of the one or more display devices, atleast one camera, a microphone array having a predetermined physicalrelationship with respect to the camera, and a controller. Thecontroller may be configured to detect, at a microphone array having apredetermined physical relationship with respect to a camera, the audioemitted from each of one or more loudspeakers, each loudspeaker having apredetermined physical relationship with respect to at least one of oneor more display devices in a conference room. The controller may furtherbe configured to generate data representing a spatial relationshipbetween each of the one or more display devices and the camera based onthe detected audio.

Example Embodiments

With reference to FIG. 1, there is depicted a block diagram of a videoconference (e.g., teleconference) system 100 in which automaticconfiguration of the behavior of the display devices of the system 100based on spatial detection may be implemented, according to an exampleembodiment. Video conference system 100 includes video conferenceendpoints 104 operated by local users/participants 106 and configured toestablish audio-visual teleconference collaboration sessions with eachother over a communication network 110. Communication network 110 mayinclude one or more wide area networks (WANs), such as the Internet, andone or more local area networks (LANs). A conference server 102 may alsobe deployed to coordinate the routing of audio-video streams among thevideo conference endpoints.

Each video conference endpoint 104 may include at least one video camera(VC) 112, at least one display device 114, a loudspeaker (LDSPKR) 116coupled to or integrated with the display device 114, one or moremicrophones arrays (MIC) 118 coupled to or integrated with the camera112, and an endpoint controller 120 configured to control the videocamera(s) 112, at least one display device 114, the loudspeaker 116, andthe one or more microphone arrays 118. In a transmit direction,endpoints 104 capture audio/video from their local participants 106 withvideo camera 112/microphone array 118, encode the captured audio/videointo data packets, and transmit the data packets to other endpoints orto the conference server 102. In a receive direction, endpoints 104decode audio/video from data packets received from the conference server102 or other endpoints and present the audio/video to their localparticipants 106 via display device 114/loudspeaker 116.

Referring now to FIG. 2A, there is depicted an illustration of videoconference endpoint 104 deployed in a conference room 200, according toan embodiment. Video conference endpoint 104 includes a plurality ofdisplay devices 114(1)-114(4) positioned around the conference room 200.Display devices 114(1)-114(3) may be screens configured to displaycontent from video sources, while display device 114(4) may be auser-interactive digital display device (e.g., a whiteboard or touchscreen). Display devices 114(1)-114(4) may contain a camera112(1)-112(4), respectively, and a microphone array 118(1)-118(4),respectively, having a predetermined physical relationship with respectto the cameras 112(1)-112(4), respectively. In some embodiments, themicrophone arrays 118(1)-118(4) may be integrated with the cameras112(1)-112(4), respectively. Cameras 112(1)-112(4) are each operatedunder control of endpoint 104 to capture video of different views orscenes of multiple participants 106 seated around a table 202 oppositefrom or facing (i.e., in front of) the cameras 112(1)-112(4) (anddisplay devices 114(1)-114(4)). The cameras 112(1)-112(4) depicted inFIG. 2A may be only one example of many possible camera and cameracombinations that may be used, as would be appreciated by one ofordinary skill in the relevant arts having read the present description(i.e., combining two video cameras for one display device).

In some forms, the display devices may be separate from one or morecameras, and the microphone arrays may be separate from the displaydevices and one or more cameras. For example, an end user may usehis/her own display devices, and some cameras available in the marketare configured to attach to a microphone stand supporting the microphonearray. However, even in that situation, the camera will, once attach,have a known predetermined physical relationship with respect to themicrophone array. In summary, the various components of an endpoint maybe integrated together when sold, or may be configured after purchase tobe physically attached to each other so as to have a predeterminedphysical relationship. Furthermore, the loudspeakers 116(1)-116(4) mayhave a predetermined physical relationship with respect to the displaydevices 114(1)-114(4), respectively. In some embodiments, theloudspeakers 116(1)-116(4) may be integrated with the display devices114(1)-114(4), respectively. While FIG. 2A illustrates the loudspeakers116(1)-116(4) being disposed centrally on the display devices112(1)-112(4), it should be appreciated that the loudspeakers116(1)-116(4) may be disposed in any location within or around theedge/frame of the display devices 112(1)-112(4), including, but notlimited to, centrally along the bottom edge of the frame of the displaydevices 112(1)-112(4), the bottom corners of the display devices112(1)-112(4), etc. In other embodiments, the loudspeakers 116(1)-116(4)may be attached or mounted in close proximity to the display devices114(1)-114(4), respectively. Thus, the loudspeakers 116(1)-116(4) areconfigured to generate audio projected in the same directions that thedisplay devices 114(1)-114(4), respectively, display video content. Inother words, the loudspeakers 116(1)-116(4) are integrated with thedisplay devices 114(1)-114(4) such that the audio outputs generated bythe loudspeakers 116(1)-116(4) originate from approximately the samelocation in which the content of the video sources are displayed.

As depicted in the example of FIG. 2A, and as briefly explained above,microphone arrays 118(1)-118(4) are positioned adjacent to, integratedwith (or otherwise in a known predetermined physical relationship to),the cameras 112(1)-112(4), respectively. In one embodiment, microphonearrays 118(1)-118(4) may be planar microphone arrays. The combination ofthe cameras 112(1)-112(4) with the microphone arrays 118(1)-118(4),respectively, may be disposed adjacent to display devices 114(1)-114(4),respectively, enabling the respective microphone arrays 118(1)-118(4) toreceive both audio from participants 106 in room 200 and the audiooutputs generated by the loudspeakers 116(1)-116(4) of display devices114(1)-114(4). Each of cameras 112(1)-112(4) may include pan, tilt, andzoom (PTZ) features that may be implemented mechanically and/ordigitally.

The video conference endpoint 104 further includes an endpoint usercontrol device 204 disposed within the conference room 200. The endpointuser control device 204 may be movable within the room 200. The endpointuser control device 204 may be a tablet computer, smartphone or othersimilar device on which an endpoint controller application is installed.The endpoint user control device 204 may be configured to manage each ofthe display devices 114(1)-114(4), including, but not limited to, thecontent displayed on each of the display devices 114(1)-114(4). Theendpoint user control device 204 may also be configured control the pan,tilt, and zoom the video cameras 112(1)-112(4) (in the mechanical ordigital domain) as necessary to capture video of different views thatencompass one or more of participants 106.

Video conference endpoint 104 uses (i) audio detection techniques todetect audio sources, i.e., loudspeakers 116(1)-116(4), by themicrophone arrays 118(1)-118(4) and to determine the spatialrelationship between the cameras 112(1)-112(4), display devices114(1)-114(4), loudspeakers 116(1)-116(4), and microphone arrays118(1)-118(4); (ii) face detection techniques to detect faces andassociated positions thereof of participants 106 around the table 202;and (iii) object detection techniques to detected the shape of specificand known objects, e.g., the table 202.

In accordance with techniques presented herein, video conferenceendpoint 104 defines/establishes the spatial relationship betweencameras 112(1)-112(4) and display devices 114(1)-114(4), andautomatically determines which display device 114(1)-114(4) will displaycertain video feeds received by the video conference endpoint 104. Insupport of this, video conference endpoint 104 also defines theprobability that an audio source detected by the microphone array118(1)-118(4) is disposed above, below, to the right of, or to the leftof the respective cameras 112(1)-112(4) and the respective microphonearrays 118(1)-118(4), and thus also defines the probability that adisplay device 114(1)-114(4) is disposed above, below, to the right of,or to the left of the respective cameras 112(1)-112(4) and respectivemicrophone arrays 118(1)-118(4). In certain cases described below,endpoint 104 automatically determines which display device 114(1)-114(4)to display a live video feed of remote participants 106 located at aremote video conference endpoint 104.

Referring now to FIG. 2B, there is depicted an illustration of a firstvideo conference endpoint 104(1) deployed in conference room 200(1) anda second video conference endpoint 104(2) deployed in conference room200(2), the two conference endpoints 104(1), 140(2) configured tocommunicate with one another via network 110, according to anembodiment. The first video conference endpoint 104(1) and the secondvideo conference endpoint 104(2) are substantially similar to the videoconference endpoint 104 depicted in FIG. 2A.

The first video conference endpoint 104(1) includes a plurality ofdisplay devices 114(1)-114(4) positioned around the conference room200(1). Display devices 114(1)-114(3) may be screens configured todisplay content from video sources, while display device 114(4) may be auser-interactive digital display device (e.g., a whiteboard or touchscreen). Display devices 114(1)-114(4) may contain a camera112(1)-112(4), respectively, and a microphone array 118(1)-118(4),respectively, integrated with the cameras 112(1)-112(4), respectively.Cameras 112(1)-112(4) are each operated under control of endpoint 104(1)to capture video of different views or scenes of multiple participants106 seated around a table 202(1) opposite from or facing (i.e., in frontof) the cameras 112(1)-112(4) (and display devices 114(1)-114(4)).Furthermore, display devices 114(1)-114(4) may contain an integratedloudspeaker 116(1)-116(4), respectively.

The second video conference endpoint 104(2) includes a plurality ofdisplay devices 114(5)-114(8) positioned around the conference room200(2). Display devices 114(5)-114(7) may be screens configured todisplay content from video sources, while display device 114(8) may be auser-interactive digital display device (e.g., a whiteboard or touchscreen). Display devices 114(5)-114(8) may contain a camera112(5)-112(8), respectively, and a microphone array 118(5)-118(8),respectively, integrated with the cameras 112(5)-112(8), respectively.Cameras 112(5)-112(8) are each operated under control of endpoint 104(2)to capture video of different views or scenes of multiple participants106 seated around a table 202(2) opposite from or facing (i.e., in frontof) the cameras 112(5)-112(8) (and display devices 114(5)-114(8)).Furthermore, display devices 114(5)-114(8) may contain an integratedloudspeaker 116(5)-116(8), respectively.

As illustrated in FIG. 2B, the first video conference endpoint 104(1)and the second video conference endpoint 104(2) are configured tocommunicate with each other via network 110. The captured video andaudio of the first video conference endpoint 104(1) may be sent to thesecond video conference endpoint 104(2), where the captured video andaudio from the first video conference endpoint 104(1) may be output bythe display devices 114(5)-114(8) and the loudspeakers 116(5)-116(8) ofthe second video conference endpoint 104(2). Conversely, the capturedvideo and audio of the second video conference endpoint 104(2) may besent to the first video conference endpoint 104(1), where the capturedvideo and audio from the second video conference endpoint 104(2) may beoutput by the display devices 114(1)-114(4) and the loudspeakers116(1)-116(4) of the first video conference endpoint 104(1).

As described herein, the video conference endpoint 104(1) may beconfigured to use data representing the spatial relationship of videoconference components generated according to the techniques presentedherein to assign video sources contained in an incoming video feedreceived from video conference endpoint 104(2) to display devices inconference room 200(1), and to assign outputs from a plurality ofcameras in conference room 200(1) in an outgoing video feed to be sentto video conference endpoint 104(2) in conference room 200(2).Similarly, video conference endpoint 104(2) may be configured to usedata representing the spatial relationship of video conferencecomponents generated according to the techniques presented herein toassign video sources contained in an incoming video feed received fromvideo conference endpoint 104(1) to display devices in conference room200(2), and to assign outputs from a plurality of cameras in conferenceroom 200(2) in an outgoing video feed to be sent to video conferenceendpoint 104(2) in conference room 200(1).

With reference to FIGS. 3A-3D, depicted is a front view of a displaydevice 114 with the camera 112 and microphone array 118 disposed atvarious positions around the display device 114. As previouslyexplained, the display device 114 includes a loudspeaker 116 integratedwith, coupled to, or mounted in close proximity with the display device114. In the examples illustrated in FIGS. 3A-3D, the loudspeaker 116 isintegrated with the display device 114 such that the loudspeaker 116 maybe disposed within the display device 114. While FIGS. 3A-3D illustratethe loudspeaker 116 being disposed centrally on the display device 112,it should be appreciated that the loudspeaker 116 may be disposed in anylocation within or around the edge/frame of the display device 112,including, but not limited to, centrally along the bottom edge of theframe of the display device 112, the bottom corners of the displaydevice 112, etc. Furthermore, the display device 114 includes a top side300, a bottom side 302 opposite the top side 300, a first or left side304, and a second or right side 306 opposite the left side 304. Thedisplay device 114 further includes a screen 310, which is configured todisplay first video content 312 and second video content 314. In oneembodiment, first video content 312 may be a presentation (document,slides, etc.), while second video content 314 may be a live video feedof remote participants 106 located at another video conference endpoint104.

As illustrated in FIG. 3A, when the camera 112 and integrated microphonearray 118 are disposed on or proximate to the top side 300 of thedisplay device 114, the video conference endpoint 104 displays the livevideo feed 314 on the screen 310 of the display device 114 proximate tothe top side 300 and the camera 112. FIG. 3B illustrates that when thecamera 112 and integrated microphone array 118 are disposed on orproximate to the bottom side 302 of the display device 114, the videoconference endpoint 104 displays the live video feed 314 on the screen310 of the display device 114 proximate to the bottom side 302 and thecamera 112.

Furthermore, FIG. 3C illustrates that when the camera 112 and integratedmicrophone array 118 are disposed on or proximate to the right side 306of the display device 114, the video conference endpoint 104 displaysthe live video feed 314 on the screen 310 of the display device 114proximate to the right side 306 and the camera 112. As illustrated inFIG. 3D, when the camera 112 and integrated microphone array 118 aredisposed on or proximate to the left side 304 of the display device 114,the video conference endpoint 104 displays the live video feed 314 onthe screen 310 of the display device 114 proximate to the top side 300and the camera 112. Thus, as illustrated in FIGS. 3A-3D, the live videofeed 314 of participants 106 from another endpoint 104 are presented onthe screen 310 of the display device 114 such that the live video feed314 is proximate to the camera 112 attached or coupled to the displaydevice 114. Positioning the live video feed 314 proximate to the camera112 enables better “eye contact” between participants 106 at differentendpoints 104. The positioning of the live video feed 314 on the screen310 of the display device 114, as described above, gives the appearancethat participants 106 at a first endpoint 104 are looking into thecamera 112 while actually viewing the live video feed 314 disposed onthe screen 310 of the display device 114.

With reference to FIGS. 4A and 4B, depicted is a front view of twodisplay devices 114(1), 114(2) arranged proximate to each other, withthe camera 112 and microphone array 118 disposed at various positionswith respect to the display devices 114(1), 114(2). Similar to theexamples illustrated in FIGS. 3A-3D, the display devices 114(1), 114(2)includes a loudspeaker 116(1), 116(2) integrated with, coupled to, ormounted in close proximity with the display devices 114(1), 114(2). Inthe examples illustrated in FIGS. 4A and 4B, the loudspeakers 116(1),116(2) are integrated with the display devices 114(1), 114(2),respectively, such that the loudspeakers 116(1), 116(2) may be disposedwithin the display devices 114(1), 114(2), respectively. As previouslyexplained, while FIGS. 4A and 4B illustrate the loudspeakers 116(1),116(2) being disposed centrally on the display devices 112(1), 112(2),it should be appreciated that the loudspeakers 116(1), 116(2) may bedisposed in any location within or around the edge/frame of the displaydevices 112(1), 112(2), including, but not limited to, centrally alongthe bottom edge of the frame of the display devices 112(1), 112(2), thebottom corners of the display devices 112(1), 112(2), etc. Furthermore,each of the display devices 114(1), 114(2) includes a top side 300(1),300(2), a bottom side 302(1), 302(2) opposite the top side 300(1),300(2), a first or left side 304(1), 304(2), and a second or right side306(1), 306(2) opposite the left side 304(1), 304(2). The displaydevices 114(1), 114(2) further include a screen 310(1), 310(2), whichare configured to display first video content 312(1), 312(4), and whichmay be capable of displaying second video content 314.

Even with multiple display devices 114(1), 114(2), the video conferenceendpoint 104 is configured to determine on which screen 310(1), 310(2)to display the second video content or live video feed 314, as well asthe positioning on the selected screen 310(1), 310(2) such that the livevideo feed 314 is positioned proximate to the camera 112 to enablebetter “eye contact” between participants 106 at different endpoints104. As illustrated in FIG. 4A, the camera 112 and integrated microphonearray 118 are disposed on or proximate to the top side 300(1) of thedisplay device 114(1). Thus, the video conference endpoint 104configures the screens 310(1), 310(2) of the display devices 114(1),114(2) to position the live video feed 314 on the screen 310(1) of thedisplay device 114(1) so that the live video feed 314 is proximate tothe top side 300(1) and to the camera 112. As illustrated, while thelive video feed 314 is configured to share the screen 310(1) of thedisplay device 114(1) with the presentation 312(1), the presentation312(2) is also configured to encompass the entire screen 310(2) of thedisplay device 114(2). Therefore, participants 106 at the endpoint 104may be able to view the content of the presentation 312(1), 312(2) oneither screen 310(1), 310(2) of either display device 114(1), 114(2),while also viewing the live video feed 314 on the screen 310(1) of thedisplay device 114(1). Because the camera 112 is disposed on the topside 300(1) of the display device 114(1), when participants 106 view thelive video feed 314 displayed on the screen 310(1) of the display device114(1), which is proximate to the top side 300(1) of the display device114(1), the participants 106 appear to also be looking into the camera112.

As illustrated in FIG. 4B, the camera 112 and integrated microphonearray 118 are disposed between the right side 306(1) of display device114(1) and the left side 304(2) of display device 114(2). In thisillustrated example, the camera 112 and integrated microphone array 118may be disposed equidistant from the right side 306(1) of display device114(1) and left side 304(2) of display device 114(2). When the camera112 and integrated microphone array 118 are disposed equidistant fromthe right side 306(1) of display device 114(1) and left side 304(2) ofdisplay device 114(2), the video conference endpoint 104 may select onwhich screen 310(1), 310(2) to display the live video feed 314. If thecamera 112 and integrated microphone array 118 are disposed between thedisplay devices 114(1), 114(2) such that the camera 112 and integratedmicrophone array 118 are closer to one of the right side 306(1) ofdisplay device 114(1) or the left side 304(2) of display device 114(2),the video conference endpoint 104 may display the live video feed 314 onthe screen 310(1), 310(2) to which the camera 112 is closest.

As FIG. 4B illustrates, the camera 112 and integrated microphone array118 are disposed between the right side 306(1) of display device 114(1)and left side 304(2) of display device 114(2), and the camera 112 andintegrated microphone array 118 are also disposed proximate to thebottom sides 302(1), 302(2) of display device 114(1), 114(2). Thus, asillustrated, the video conference endpoint 104 displays the live videofeed 314 in the bottom right corner of the screen 310(1) of the displaydevice 114(1) proximate to both the bottom side 302(1) and the rightside 306(1) of the display device 114(1). As previously explained, whilethe live video feed 314 is configured to share the screen 310(1) of thedisplay device 114(1) with the presentation 312(1), the presentation312(2) is configured to encompass the entire screen 310(2) of thedisplay device 114(2). Therefore, participants 106 at the endpoint 104may be able to view the content of the presentation 312(1), 312(2) oneither screen 310(1), 310(2) of either display device 114(1), 114(2),while also viewing the live video feed 314 on the screen 310(1) of thedisplay device 114(1).

Reference is now made to FIG. 5, which shows an example block diagram ofan endpoint controller 120 of video conference endpoint 104 configuredto perform techniques described herein. There are numerous possibleconfigurations for endpoint controller 120 and FIG. 5 is meant to be anexample. Endpoint controller 120 includes a processor 500, a networkinterface unit (NIU) 502, and memory 504. The network interface (I/F)unit (NIU) 502 is, for example, an Ethernet card or other interfacedevice that allows the endpoint 104 to communicate over communicationnetwork 110 (FIG. 1). Network interface unit 502 may include wiredand/or wireless connection capability.

Processor 500 may take the form of a collection of microcontrollersand/or microprocessors, for example, each configured to executerespective software instructions stored in the memory 504. Thecollection of microcontrollers may include, for example: a videocontroller to receive, send, and process video signals related todisplay 114 and video camera 112; an audio processor to receive, send,and process audio signals related to loudspeaker 116 and microphonearray 118; and a high-level controller to provide overall control.Portions of memory 504 (and the instruction therein) may be integratedwith processor 500. As used herein, the terms “audio” and “sound” aresynonymous and interchangeable.

In a distributed processor embodiment, endpoint controller 120 is adistributed processor, including, but not limited to, (i) an audioprocessor for the microphone array 118 to determine audio angle ofarrival of a sound source (as discussed below), and (ii) a videocoder/decoder (i.e., codec) that is also configured to analyze thecontent of the video sources received by the endpoint 104.

The memory 504 may include read only memory (ROM), random access memory(RAM), magnetic disk storage media devices, optical storage mediadevices, flash memory devices, electrical, optical, or otherphysical/tangible (e.g., non-transitory) memory storage devices. Thus,in general, the memory 504 may comprise one or more computer readablestorage media (e.g., a memory device) encoded with software comprisingcomputer executable instructions and when the software is executed (bythe processor 500) it is operable to perform the operations describedherein. For example, the memory 504 stores or is encoded with softwareinstructions for Source Display Positioning Module 506 to performoperations described herein for determining the spatial relationshipbetween camera 112 and display devices 114 and determining which displaydevice 114 will display a live video feed. Source Display PositioningModule 506 also includes an Audio Analysis Module 508 and an ImageAnalysis Module 510. Audio Analysis Module 508 may determine the angleof arrival of a sound source as received by the microphone array 118.Image Analysis Module 510 may evaluate the content of video sourcesreceived by the video conference endpoint 104 and determine whichdisplay device 114 will display a received video source based on theinformation acquired by the Audio Analysis Module 508.

With reference to FIG. 6, depicted is an example of endpoint usercontrol device 204. The endpoint user control device 204 may have adisplay 600, such as a touchscreen display. The display 600 of theendpoint user control device 204 may be configured to presentrectangular representations 602(1)-602(3) of the display devices114(1)-114(3) operable at the endpoint 104. Thus, the display 600 of theendpoint user control device 204 may display a three dimensionalrepresentation of the conference room 200. As illustrated in the exampleof FIG. 6, two display representations 602(1), 602(2) are disposed nextto each other and are facing the third display representation 602(3).Thus, the conference room 200 may contain two display devices 114(1),114(2) on one side of the room 200 and a third display device 114(3) onan opposite side of the room 200, where the third display device 114(3)faces the other two display devices 114(1), 114(2). A user may touch ortap the touch screen display 600 of the endpoint user control device 204at the location of one of the display representations 602(1)-602(3) tocontrol (e.g., display content, volume control, etc.) the display device114(1)-114(3) represented by the selected display representation602(1)-602(3). Furthermore, the display 600 of the endpoint user controldevice 204 may further present other controls and functions 604 at thebottom of the display 600.

With reference to FIGS. 7A and 7B, depicted are audio outputs of aloudspeaker 116 being detected by the microphone array 118 integratedwith a camera 112 along a horizontal plane (FIG. 7A) and a verticalplane (FIG. 7B). The microphone array 118 detects audio outputs by theloudspeaker 116 and determines relative angles of the loudspeaker 116originating the audio output with reference, or in relation to, thedirection A in which the camera is facing (e.g., a normal of the camera112). As illustrated in FIG. 7A, audio outputs detected by themicrophone array 118 and originating from a loudspeaker 116 disposed tothe right of the camera 112 may be given an angular measurement of θ,while audio outputs detected by the microphone array 118 and originatingfrom a loudspeaker 116 disposed to the left of the camera 112 may begiven an angular measurement of −θ. Thus, the angular measurements of θand −θ represent the azimuth angles of the detected audio output withrespect to the normal A of the camera 112. As illustrated in FIG. 7B,audio outputs detected by the microphone array 118 and originating froma loudspeaker 116 disposed above the camera 112 may be given an angularmeasurement of φ, while audio outputs detected by the microphone array118 and originating from a loudspeaker 116 disposed below the camera 112may be given an angular measurement of −φ. Thus, the angularmeasurements of φ and −φ represent the elevation angles of the detectedaudio output with respect to the normal A of the camera 112.

With reference to FIG. 8 and continued reference to FIGS. 3A-3D, 4A, 4B,5, 7A, and 7B, there is depicted a flowchart of an example method 800 ofdetermining the spatial relationship between a display device 114 withan integrated loudspeaker 116 and a camera 112 with an integratedmicrophone array 118 based on the audio generated by the loudspeaker116. Initially, at 805, the microphone array 118 receives a new frame ofaudio samples from a loudspeaker 116. The endpoint controller 120 may beconfigured to cause the loudspeaker 116 to generate an audio output andthe microphone array 118 is configured to detect the audio output. At810, the endpoint controller 120 begins determining the azimuth (θ) andelevation (φ) angles to the loudspeaker 116 from the microphone array118, while also triangulating a distance (r) from the loudspeaker 116generating the audio outputs detected by the microphone array 118.

At 815, the endpoint controller 120 determines whether the audio outputdetected by the microphone array 118 originates from a location (e.g., aloudspeaker 116) that is less than a predetermined distance (e.g., threemeters) away from the microphone array 118. If it is determined at 815that the loudspeaker 116 is less than the predetermined distance (threemeters) away from the microphone array 118, then the endpoint controller120 continues to determine the azimuth and elevation angles of thedetected audio output with respect to the microphone array 118 at 820and 850. However, if, at 815, the endpoint controller 120 determinesthat the detected audio output is not less than the predetermineddistance (three meters) away from the microphone array 118, then theendpoint controller 120 skips determining the azimuth and elevationangles of the detected audio, and, at 895, does not update the picturein picture positioning of the live video feed 314. When the detectedaudio output originates more than the predetermined distance (threemeters) from the microphone array 118, the positioning of the live videofeed 314 may not be updated because the live video feed 314 may alreadybe disposed in an optimized position. However, other examples ofdetected audio outputs that originated from more than three meters fromthe microphone array 118 could include audio outputs that originate fromexternal sources (e.g., talking participants 106, participant 106devices, etc.), detected the audio outputs that originated from aloudspeaker 116 but reflected off of the floor and/or walls, etc.

After determining that the detected audio output originates from alocation less than the predetermined distance (three meters) away fromthe microphone array 118 (e.g., from a loudspeaker 116 disposed lessthan three meters from the microphone array 118), then at 820, theendpoint controller 120 calculates whether the audio output detected bythe microphone array has an elevation angle φ greater than 75 degrees.If, at 820, the determined elevation angle φ is greater than 75 degrees,then, at 825, the endpoint controller 120 increases a probability valuethat the loudspeaker 116 is above the camera 112 (P(above)). If, at 820,the determined elevation angle φ is not greater than 75 degrees, then,at 830, the endpoint controller 120 decreases the probability that theloudspeaker 116 is above the camera 112 (P(above)).

If the endpoint controller 120 decreases the probability that theloudspeaker 116 is above the camera (P(above)), then, at 835, theendpoint controller 120 calculates whether the audio output detected bythe microphone array has an elevation angle φ less than −75 degrees. If,at 835, the determined elevation angle φ is less than −75 degrees, then,at 840, the endpoint controller 120 increases the probability that theloudspeaker 116 is below the camera 112 (P(below)). If, at 835, thedetermined elevation angle φ is not less than −75 degrees, then, at 845,the endpoint controller 120 decreases the probability that theloudspeaker 116 is below the camera 112 (P(below)).

At 850, the endpoint controller 120 calculates whether the audio outputdetected by the microphone array has an azimuth angle θ greater than 75degrees. If, at 850, the determined azimuth angle θ is greater than 75degrees, then, at 855, the endpoint controller 120 increases theprobability that the loudspeaker 116 is disposed to the right of thecamera 112 (P(right)). If, at 850, the determined azimuth angle θ is notgreater than 75 degrees, then, at 860, the endpoint controller 120decreases the probability that the loudspeaker 116 is disposed to theright of the camera 112 (P(right)).

If the endpoint controller 120 decreases the probability that theloudspeaker 116 is disposed to the right of the camera 112 (P(right)),then, at 865, the endpoint controller 120 calculates whether the audiooutput detected by the microphone array 118 has an azimuth angle θ lessthan −75 degrees. If, at 865, the determined azimuth angle θ is lessthan −75 degrees, then, at 870, the endpoint controller 120 increasesthe probability that the loudspeaker 116 is disposed to the left of thecamera 112 (P(left)). If, at 865, the determined azimuth angle θ is notless than −75 degrees, then, at 875, the endpoint controller 120decreases the probability that the loudspeaker 116 is disposed to theleft of the camera 112 (P(left)).

After all of the probabilities P(above), P(below), P(right), P(left)have been calculated and it is verified that the loudspeaker 116 is lessthan 3 meters away from the microphone array 118, the endpointcontroller 120, at 880, determines a spatial relationship value Sbetween the loudspeaker 116 generating the audio output and themicrophone array 118 by determining which calculated probabilityP(above), P(below), P(right), P(left) has the largest value. In oneembodiment, the endpoint controller 120 may also disregard any of theprobabilities P(above), P(below), P(right), P(left) with lower values.At 885, the endpoint controller 120 then determines whether or not thespatial relationship value S is greater than a predetermined thresholdvalue. If, at 885, the endpoint controller 120 determines that thespatial relationship value S is greater than the predefined threshold,then, at 890, the endpoint controller 120 updates the picture in picturepositioning of the live video feed 314 so that the live video feed 314is positioned proximate to the camera 112 as illustrated in FIGS. 3A-3D,4A, and 4B. However, if, at 885, the endpoint controller 120 determinesthat the spatial relationship value S is less than the predefinedthreshold, then, at 895, the endpoint controller 120 does not update thepicture in picture positioning of the live video feed 314 because thelive video feed 314 may already be disposed in an optimized positionproximate to the camera 112, as illustrated in FIGS. 3A-3D, 4A, and 4B.

In another example, with reference to the conference rooms 200, 200(1),and 200(2) depicted in FIGS. 2A and 2B, and with continued reference toFIGS. 7A and 7B, the endpoint controller 120 is configured to determinewhich of cameras 112(1)-112(3) is best oriented to capture a participant106 presenting or collaborating on the user-interactive display device114(4). In this example, the endpoint controller 120 may utilize themicrophone arrays 118(1)-118(3) integrated with the cameras112(1)-112(3) to detect audio outputs from the loudspeaker 116(4) ofuser-interactive display device 114(4), which is configured as awhiteboard or other similar presentation/display device, in order todetermine the spatial relationship between the cameras 112(1)-112(3) andthe user-interactive display device 114(4). The endpoint controller 120may, from the detected audio output, calculate the azimuth and elevationangles, as well as the distance, of the loudspeaker 116(4) with respectto the each of the microphone arrays 118(1)-118(3). As previouslyexplained, because the loudspeaker 116(4) is integrated with theuser-interactive display device 114(4) and because the microphone arrays118(1)-118(3) are integrated with the respective cameras 112(1)-112(3),the calculated distance and azimuth and elevation angles also representthe spatial relationship of the user-interactive display device 114(4)with respect to the respective cameras 112(1)-112(3). However, unlikethe example method of FIG. 8, where the endpoint controller 120 makesits determination on which display device 114(1)-114(4) is best utilizedto display a video source based on a calculated short distance (e.g.,less than 3 meters from one of the cameras 112(1)-112(4)) and calculatedlarge azimuth and elevation angles (e.g., greater than 75 degrees withrespect to the normal A of one of the cameras 112(1)-112(4)), theendpoint controller 120 may assign the function of displaying thepresenting participant 106 standing at or next to the user-interactivedisplay device 114(4) to a particular one of the cameras 112(1)-112(3)that is calculated to be a large distance from the user-interactivedisplay device 114(4) and that has relatively small azimuth andelevation angles with respect to the user-interactive display device114(4). When it is calculated that user-interactive display device114(4) is greater than a predetermined distance from one of the cameras112(1)-112(3), and that the user-interactive display device 114(4) isoriented with relatively small azimuth and elevation angles (i.e., lessthan or equal to a predetermined angle) with respect to the normal A ofone of the cameras 112(1)-112(3), the endpoint controller 120 may makethe determination that a particular one of the cameras 112(1)-112(3) isboth pointed in the general direction of the user-interactive displaydevice 114(4) (e.g., the normal A of one of the cameras 112(1)-112(3)extends in the general direction of the user-interactive display device114(4)) and is disposed within the conference room 200 at a locationopposite of the user-interactive display device 114(4). As illustratedin FIG. 2A, the endpoint controller 120 of conference room 200 may makethe determination that camera 112(2) has an acceptable field of view ofthe user-interactive display device 114(4), where camera 112(2) iscapable of capturing the user-interactive display device 114(4), itemsdisplayed on the user-interactive display device 114(4), and anyparticipant that may be presenting or collaborating on theuser-interactive display device 114(4). After making this determination,the endpoint controller 120 may assign camera 112(2) with the functionof capturing the user-interactive display device 114(4) and anyparticipant that may be present at the user-interactive display device114(4) such that the field of view of the camera 112(2) can betransmitted to another video conference endpoint.

With continued reference to FIGS. 2A, 2B, and 8, once the spatialrelationship (e.g., the azimuth (θ) angles, the elevation (φ) angles,and the distance (r)) between each of the display devices 114(1)-114(4)and each of the cameras 112(1)-112(4) has been calculated, the endpointcontroller 120(1) of the first conference room 200(1) can assign inboundvideo sources (e.g., video sources received by the first conference room200(1) from the second conference room 200(2)) to the display devices114(1)-114(4) within the first conference room 200(1). The assignment ofthe inbound video sources to the display devices 114(1)-114(4) may bebased on the spatial relationship of each of the display devices114(1)-114(4) with each of the cameras 112(1)-112(4), and the respectivelocations of each of the display devices 114(1)-114(4) and each of thecameras 112(1)-112(4) within the first conference room 200(1).Similarly, the endpoint controller 120(1) may also assign video outputsfrom the cameras 112(1)-112(4) to an outgoing video feed sent from thefirst conference room 200(1) to the second conference room 200(2). Aspreviously explained, the cameras 112(1)-112(4) may be configured andoperated to capture video of different views or scenes of multipleparticipants 106 seated around a table 202(1) opposite from or facing(i.e., in front of) the cameras 112(1)-112(4) (and display devices114(1)-114(4)). The cameras 112(1)-112(4) may also be configured andoperated to capture video of participants 106 disposed around particulardisplay devices 114(1)-114(4). The assignment of the captured video maybe based on the data representing the spatial relationship of each ofthe cameras 112(1)-112(4) with each of the display devices114(1)-114(4), and the respective location of each of the cameras112(1)-112(4) within the first conference room 200(1). Each of the videooutputs may be tagged or labeled with metadata indicating the respectivefield of view of each of the cameras 112(1)-112(4) that captured each ofthe video outputs. This tagged metadata may be utilized by a remoteconference room, such as the second conference room 200(2), to furtherdetermine how to assign the inbound video sources of the secondconference room 200(2) to the display devices 114(5)-114(8) of thesecond conference room 200(2). The tagged metadata is also useful forthe remote conference rooms, such as the second conference room 200(2),when the inbound video source simultaneously includes video outputs frommore than one camera 112(1)-112(4) of the first conference room 200(1),this is referred to as multi-stream. That is, a video feed may includemultiple video streams.

With reference to FIG. 9, depicted is a flowchart of an example method900 for utilizing the loudspeakers 116 and the microphone array 118 todetermine spatial relationship between the camera 112 and the displaydevices 114 disposed within a conference room 200. Reference is alsomade to FIGS. 3A-3D for purposes of the description of FIG. 9. At 905,the endpoint controller 120 plays an audio output out of each of thedisplay devices 114 that contain a loudspeaker 116. The display devices112 may be connected to the endpoint 104 via a high-definitionmultimedia interface (HDMI) cable, which is capable of transporting bothvideo and audio signals over the same cable. In the event that an audiosignal is a multi-channel audio signal, the loudspeaker 116 integratedwith the display device 114 may output a separate audio output for eachchannel of the audio signal. At 910, the microphone array 118 integratedwith the camera 112 detects the audio outputs to determine the spatiallocation (e.g., above, below, side, opposite, etc.) of the displaydevice 114 with respect to the camera. The steps at 905 and 910 may berepeated for each of the display devices 114 and for each of the cameras112 located within the conference room 200. The detection anddetermination of the spatial relationship between the cameras 112 andthe display devices 114 may be completed as described above with respectto FIG. 8.

Once all of the spatial relationship and placement data has beencollected, the endpoint controller, at 915, builds an internal model ofthe placement and relationship of the display devices 114 and cameras112 in the conference room 200. At 920, the endpoint controller 120 thenconfigures the display device roles and rules for presenting video andaudio content based on the derived model and the content of the videoand audio sources. For example, if, based on the derived model, thecamera 112 is adjacent to (e.g., above, below, to the side) a displaydevice 114 and the content of one of the video and audio sources is alive video feed 314 of another endpoint 104, then the endpointcontroller 120 may determine that that display device 114 should receiveand display the live video feed 314. In another example, if, based onthe derived model, the camera 112 is disposed opposite of the displaydevice 114 (e.g., the camera 112 is across the conference room 200 froma user-interactive display device 114), the endpoint controller 120 maydetermine that that camera 112 should be utilized to display thepresentation presented on that display device 114. Finally, at 925, theendpoint controller 120 presents the three dimensional model to theconference room 200 through the display 600 of the endpoint user controldevice 204, illustrated in FIG. 6.

With reference to FIG. 10, illustrated is a flowchart of a method 1000for determining the spatial relationship between a single camera 112 anda single display device 114, where the camera 112 is disposed eitherabove or below the display device 114. Reference is also made to FIGS.2A, 2B, 3A-3D, 4A, 4B, 5, 8, and 11A-11C for purposes of the descriptionof FIG. 10. The method 1000 includes a plurality of processes fordetermining the location of the camera 112 with respect to the displaydevice 114. These processes include using the microphone array 118integrated with the camera 112 to determine the location of theloudspeaker 116 integrated with the display device 114, as describedabove, and detecting the faces of participants 106 seated around thetable 202 within the conference room 200. These techniques may beperformed together or in lieu of each other when the conditions for oneof the techniques are not sufficient for that technique to adequatelydetermine the positioning of the camera 112 with respect to the displaydevice 114

At 1005, the endpoint controller 120 utilizes facial detection software,in conjunction with the camera 112 of a conference room 200 of anendpoint 104, to detect the faces of participants 106 positioned arounda table 202 within a conference room 200. Facial detection techniquesare well known in the art and are not described in detail herein. At1010, the endpoint controller 120 analyzes the image captured by thecamera 112 and plots a line passing through the detected faces aroundthe table 202, as shown in FIGS. 11A-11C and described hereinafter. Theline passing through the detected faces could be a parabola, which maybe a function of the equation y=ax²+bx+c, that is fitted over thelocation of the detected faces using any conventional method including,but not limited to, the method of least squares. When the line passingthrough the detected faces is represented by the equation y=ax²+bx+c andthe value of “a” is greater than zero, than the line may have acurvature that opens upward, like that of line B illustrated in FIG.11A. If “a” has a value of zero in the equation y=ax²+bx+c, then theline represented by the equation y=ax²+bx+c may be a straight line, likethat of line C illustrated in FIG. 11B. However, when “a” in theequation y=ax²+bx+c is of a value less than zero, then the linerepresented by the equation y=ax²+bx+c may have a downward facingcurvature, like that of line D illustrated in FIG. 11C.

After the endpoint controller 120 plots a line passing through thedetected faces of the participants 106 within a conference room 200, theendpoint controller 120, at 1015, determines whether or not the value of“a” in the equation y=ax²+bx+c is greater than or equal to zero. If, at1015, the value of “a” is found to be greater than or equal to zero,then, at 1020, the endpoint controller 120 can make the determinationthat the camera 112 is disposed at a lower height within the conferenceroom 200 and is thus likely disposed below the display device 114 in theconference room 200. FIGS. 11A and 11B illustrate the field of view ofthe camera 112 when the camera 112 is mounted below the display device114. FIG. 11A illustrates the field of view of the camera 112 and thecurved line B, where the value of “a” is greater than zero. The detectedfaces of the participants 106 seated around the conference table 202 inFIG. 11A are aligned with the upwardly curved line B. FIG. 11Billustrates the field of view of the camera 112 and horizontal line C,where the value of “a” is equal to zero. The detected faces of theparticipants 106 seated around the conference table 202 in FIG. 11B arealigned with horizontal line C. The difference between line B and line Cmay demonstrate that the camera 112 in FIG. 11A is disposed at a lowerheight than the camera 112 in FIG. 11B. As the height of the camera 112increases, the curvature of the line through the detected faces of theparticipants transitions from an upwardly curved line to a downwardlycurved line. Thus, FIG. 11B illustrates a field of view of the camera112 when the camera 112 is disposed more closely to being equal inheight, or is disposed equal in height, with the display device 114 thanthat of FIG. 11A.

However, if, at 1015, the value of “a” is found to be less than zero,then, at 1025, the endpoint controller 120 can make the determinationthat the camera 112 is disposed at a higher height within the conferenceroom 200 and is thus likely disposed above the display device 114 in theconference room 200. FIG. 11C illustrates the field of view of thecamera 112 when the camera 112 is mounted above the display device 114.FIG. 11A illustrates the field of view of the camera 112 and the curvedline D, where the value of “a” is less than zero. The detected faces ofthe participants 106 seated around the conference table 202 in FIG. 11Aare aligned with the downwardly curved line D.

In some embodiments, in order to rely on the facial detection method thedetected faces of the participants 106 disposed in the middle of thefield of view of the camera 112 need to be smaller than the detectedfaces of the participants 106 faces disposed on the sides of the fieldof view of the camera 112. As illustrated in FIGS. 11A-11C, the detectedfaces disposed on the sides of the field of view of the camera 112 arelarger than those of the detected faces disposed centrally in the fieldof view of the camera 112. In addition, the least square error duringcurve fitting needs to be below a certain predetermined threshold. Ifthe least square error exceeds the predetermined threshold, the seatingarrangement around the conference room table 202 may not be conducive tothe facial detection method (e.g., a large conference room table 202 maynot be disposed in the middle of the conference room 200). In otherembodiments, the reference value by which to compare the value of “a”from the equation of y=ax²+bx+c may be greater or lesser than zero.

Returning back to FIG. 10, in addition to utilizing facial detectionsoftware at 1005, or instead of using facial detection software (e.g.,when participants 106 are not located within the conference room 200,not enough participants 106 are located within the conference room 200,etc.), then, at 1030, the endpoint controller 120 may, as describedabove with respect to FIGS. 7A, 7B, 8, and 9, track audio outputs, usingthe microphone array 118 integrated with the camera 112, to localize theposition of the loudspeaker 116 integrated with the display device 114.At 1035, the endpoint controller 120 determines, based on the audiotracking, if the camera 112 is mounted above or below the display device114.

At 1035, once the information has been collected regarding whether thecamera 112 is mounted above or below the display device 114, theendpoint controller 120 uses the information to update the layout of thescreen 310 of the display device 114. As previously explained andillustrated with regard to FIG. 3A, if it is determined that the camera112 is mounted above the display device 114, then the endpointcontroller 120 may position the live video feed 314 on the screen 310proximate to the top edge 300 of display device 114. Conversely, aspreviously explained and illustrated with regard to FIG. 3B, if it isdetermined that the camera 112 is mounted below the display device 114,then the endpoint controller 120 may position the live video feed 314 onthe screen 310 proximate to the bottom edge 302 of display device 114.

With reference to FIG. 12, illustrated is a flowchart of a method 1200performed by the endpoint controller 120 for assigning video and audiosources to the display devices 114 located within a conference room.Reference is also made to FIGS. 2A, 2B, 3A-3D, 4A-4B, and 5 for purposesof the description of FIG. 12. At 1205, the endpoint controller 120detects, at a microphone array 118 having a predetermined physicalrelationship with respect to a camera 112, the audio emitted from eachof one or more loudspeakers 116, where each of the one or moreloudspeakers 116 have a predetermined physical relationship with respectto at least one of one or more display devices 114 in a conference room.Thus, the endpoint controller 120 enables the microphone array 118 todetect the audio that is played, generated, or emitted from eachloudspeaker 116 of each display device 114 located within a conferenceroom 200. At 1210, the endpoint controller 120 utilizes knowntriangulation and audio localization algorithms to determine thedirection and distance from the microphone array 118 to each of theloudspeakers 116 that output audio received by the microphone array 118.In other words, from the detected audio, the endpoint controller 120 maydetermine the spatial relationship between the microphone array 118 andthe loudspeakers 116 within a conference room 200. Because, aspreviously explained, the microphone array 118 has a known predeterminedphysical relationship with respect to the camera 112 and each of theloudspeakers 116 has a known predetermined physical relationship withrespect to the one of the display devices 114, determining the spatialrelationship between the microphone array 118 and the loudspeakers 116also determines the spatial relationship between the camera 112 and thedisplay devices 114.

After determining the spatial relationship between the camera 112 andthe display devices 114, the endpoint controller 120 may then assignvideo sources to each of the plurality of display devices based on thedata representing the spatial relationship, the content of the videosources, and the use of the camera 112. For example, if it is determinedthat a camera is disposed adjacent to a display device 114 (e.g.,mounted directly above or directly below the display device 114), thenthat display device 114 may receive a live video feed of another remotevideo conference endpoint operated by other participants 106 while thecamera 112 records a live video feed of the conference room 200 in whichit is disposed. The live video feed of the conference room 200 may besent to the remote video conference endpoint for viewing by participantsat that remote video conference endpoint. In another example, aspreviously explained, if it is determined that another display device,such as a user-interactive display device, is disposed opposite of acamera 112, that camera 112 may be used to show the participant 106presenting or collaborating on the user-interactive display device 114.

Techniques presented herein automatically determine the roles of thecameras and the display devices of a video conference endpoint within aconference room when participating in a video conference. The detectionof the placement of components of a video conference endpoint/system ina room is automated using spatial detection of audio signals emitted bythe components using a microphone array and image analysis to optimizescreen usage and visualization of the room for simpler control. Noadditional equipment is needed. Rather, the equipment to be installed aspart of the video conference endpoint is used for the process. Thismakes it easy to add and remove components, such as a digital whiteboardto/from that room. In addition to automatic setup, these techniques canbe useful in suggesting layouts and room configurations in a semi-manualwizard-guided type of setup procedure.

Specifically, generated audio from the each of the loudspeakers of theone or more display devices is detected by the microphone arrays of eachof the cameras to determine the spatial relationship between each of thecameras and each of the loudspeakers within a conference room. Thedetermined spatial relationship, along with the content of the video andaudio sources of the video conference may be used by the controller ofthe video conference endpoint to automatically determine the roles ofthe one or more display devices and each of the cameras of theconference room. This eliminates a need to manually set up each videoconference room, and eliminates the need to have participants of thevideo conference manually switch the roles of the display devices andthe cameras during a video conference.

In summary, in one form, a method is provided comprising: detecting, ata microphone array having a predetermined physical relationship withrespect to a camera, the audio emitted from each of one or moreloudspeakers, each loudspeaker having a predetermined physicalrelationship with respect to at least one of one or more display devicesin a conference room; and generating data representing a spatialrelationship between each of the one or more display devices and thecamera based on the detected audio.

In another form, an apparatus is provided comprising: a cameraconfigured to capture video of a field of view; a microphone arrayhaving a predetermined physical relationship with respect to the camera,the microphone array configured to transduce audio received at themicrophone array; and a processor to control the camera and themicrophone array to: cause the microphone array to detect audio emittedfrom one or more loudspeakers having a predetermined physicalrelationship with respect to at least one of one or more display devicesin a conference room; and generate data representing a spatialrelationship between each of the one or more display devices and thecamera based on the detected audio.

In yet another form, a (non-transitory) processor readable medium isprovided. The medium stores instructions that, when executed by aprocessor, cause the processor to: detect, at a microphone array havinga predetermined physical relationship with respect to a camera, theaudio emitted from each of one or more loudspeakers, each loudspeakerhaving a predetermined physical relationship with respect to at leastone of one or more display devices in a conference room; and generatedata representing a spatial relationship between each of the one or moredisplay devices and the camera based on the detected audio.

As described herein, the data representing the spatial relationship maybe used to assign one or more video sources of an incoming video feedfrom a remote conference room to corresponding ones of the one or moredisplay devices. Similarly, the data representing the spatialrelationship may be used to assign video outputs from a plurality ofcameras in a conference room to an outgoing video feed to be sent to aremote conference room.

The above description is intended by way of example only. Variousmodifications and structural changes may be made therein withoutdeparting from the scope of the concepts described herein and within thescope and range of equivalents of the claims.

What is claimed is:
 1. A method comprising: detecting, at a microphonearray having a predetermined physical relationship with respect to acamera, audio emitted from each of one or more loudspeakers, eachloudspeaker having a predetermined physical relationship with respect toat least one of a plurality of display devices in a conference room;generating data representing a spatial relationship between each of theplurality of display devices and the camera based on the detected audio,wherein generating data representing the spatial relationship includesdetermining azimuth and elevation angles between the camera and each ofthe plurality of display devices; and assigning one or more videosources of an incoming video feed from a remote conference room tocorresponding ones of the plurality of display devices based on the datarepresenting the spatial relationship and content of the one or morevideo sources.
 2. The method of claim 1, wherein generating datarepresenting the spatial relationship further comprises: determining,based on the determined azimuth and elevation angles, at least one of: afirst probability that the camera is disposed above one of the displaydevices, a second probability that the camera is disposed below one ofthe display devices, a third probability that the camera is disposedright of one of the display devices, and fourth probability that thecamera is disposed left of one of the display devices.
 3. The method ofclaim 2, wherein generating data representing the spatial relationshipfurther comprises: determining a spatial relationship value bydetermining which of the first probability, the second probability, thethird probability, and the fourth probability contains a largestprobability value.
 4. The method of claim 3, wherein assigningcomprises: assigning a video source of the incoming video feed to bedisplayed on a top side of a screen, a bottom side of the screen, aright side of the screen, or a left side of the screen of one of thedisplay devices based on the spatial relationship value.
 5. The methodof claim 1, further comprising: determining, from the detected audio ofeach of the one or more loudspeakers, whether each of the one or moreloudspeakers is within a predetermined distance from the microphonearray.
 6. The method of claim 1, wherein the microphone array isintegrated with the camera, and each of the one or more loudspeakers isintegrated with one of the plurality of display devices.
 7. An apparatuscomprising: a camera configured to capture video of a field of view; amicrophone array having a predetermined physical relationship withrespect to the camera, the microphone array configured to transduceaudio received at the microphone array; and a processor configured tocontrol the camera and the microphone array, wherein the processor isconfigured to: cause the microphone array to detect audio emitted fromeach of one or more loudspeakers, each loudspeaker having apredetermined physical relationship with respect to at least one of aplurality of display devices in a conference room; generate datarepresenting a spatial relationship between each of the plurality ofdisplay devices and the camera based on the detected audio bydetermining azimuth and elevation angles between the camera and each ofthe plurality of display devices; and assign one or more video sourcesof an incoming video feed from a remote conference room to correspondingones of the plurality of display devices based on the data representingthe spatial relationship and content of the one or more video sources.8. The apparatus of claim 7, wherein the processor, when generating datarepresenting the spatial relationship, is further configured to:determine, based on the determined azimuth and elevation angles, atleast one of: a first probability that the camera is disposed above oneof the display devices, a second probability that the camera is disposedbelow one of the display devices, a third probability that the camera isdisposed right of one of the display devices, and a fourth probabilitythat the camera is disposed left of one of the display devices.
 9. Theapparatus of claim 8, wherein the processor, when generating datarepresenting the spatial relationship, is further configured to:determine a spatial relationship value by determining which of the firstprobability, the second probability, the third probability, and thefourth probability contains a largest probability value.
 10. Theapparatus of claim 9, wherein the processor is further configured to:assign a video source of the incoming video feed to be displayed on atop side of a screen, a bottom side of the screen, a right side of thescreen, or a left side of the screen of one of the display devices basedon the spatial relationship value.
 11. The apparatus of claim 7, whereinthe processor is further configured to: determine, from the detectedaudio of each of the one or more loudspeakers, whether each of the oneor more loudspeakers is within a predetermined distance from themicrophone array.
 12. The apparatus of claim 7, wherein the microphonearray is integrated with the camera, and each of the one or moreloudspeakers is integrated with one of the plurality of display devices.13. One or more non-transitory computer readable storage media, thecomputer readable storage media being encoded with software comprisingcomputer executable instructions, and when the software is executed,operable to: detect, at a microphone array having a predeterminedphysical relationship with respect to a camera, audio emitted from eachof one or more loudspeakers, each loudspeaker having a predeterminedphysical relationship with respect to at least one of a plurality ofdisplay devices in a conference room; generate data representing aspatial relationship between each of the plurality of display devicesand the camera based on the detected audio by determining azimuth andelevation angles between the camera and each of the plurality of displaydevices; and assign one or more video sources of an incoming video feedfrom a remote conference room to corresponding ones of the plurality ofdisplay devices based on the data representing the spatial relationshipand content of the one or more video sources.
 14. The non-transitorycomputer readable storage media of claim 13, wherein the instructionsare further operable to: determine, based on the determined azimuth andelevation angles, at least one of: a first probability that the camerais disposed above one of the display devices, a second probability thatthe camera is disposed below one of the display devices, a thirdprobability that the camera is disposed right of one of the displaydevices, and a fourth probability that the camera is disposed left ofone of the display devices.
 15. The non-transitory computer readablestorage media of claim 14, wherein the instructions are further operableto: determine a spatial relationship value by determining which of thefirst probability, the second probability, the third probability, andthe fourth probability contains a largest probability value.
 16. Thenon-transitory computer readable storage media of claim 15, wherein theinstructions are further operable to: assign a video source of theincoming video feed to be displayed on a top side of a screen, a bottomside of the screen, a right side of the screen, or a left side of thescreen of one of the display devices based on the spatial relationshipvalue.
 17. The non-transitory computer readable storage media of claim13, wherein the instructions are further operable to: determine, fromthe detected audio of each of the one or more loudspeakers, whether eachof the one or more loudspeakers is within a predetermined distance fromthe microphone array.
 18. The method of claim 1, wherein the incomingvideo feed and the content are displayed simultaneously on a displaydevice of the plurality of display devices.
 19. The apparatus of claim7, wherein the incoming video feed and the content are displayedsimultaneously on a display device of the plurality of display devices.20. The non-transitory computer readable storage media of claim 13,wherein the incoming video feed and the content are displayedsimultaneously on a display device of the plurality of display devices.